A Systematic Literature Review Model for Religion

Traditional literature reviews (TLR) are found in all disciplines including religion. In TLR a research question is chosen, resources are identified that have information relevant to the question, and one uses information from the resources to address the TLR question. Such reviews give writers and readers background knowledge, help them get up to speed, or help to set the stage for further research. In academia, projects supported by TLR include student papers, theses and dissertations, grants, and articles. In a religious community, sermons and religious instruction might be served by TLR. The information explosion is ongoing, and so TLR clearly have value. In addition, however, in the health and social sciences the "systematic review" (SR) is now widely viewed as an important alternative for those doing and reading literature reviews. Two reasons are frequently given for why the conclusions in SR receive special attention: 1. comprehensive literature searching is used (important for limiting bias), and 2. detailed/transparent reporting of methods are used (making replication and evaluation of methods possible). SR can increase confidence in conclusions of reviews. A part of the first author’s research on SR, philosophy, and religion involves exploring the possibilities of SR for research in religion. A “scoping review” (type of SR) examined the literature on TLR and SR methods in and for religion. Although TLR were found in the religion literature, SR were not. Drawing on significant SR methods literature, a SR model was developed that could provide a more rigorous framework for literature reviews in religion. A Systematic Review Model for Relgion 3 Traditional literature reviews are writings that ‘pull together information from what has been published or said about a topic’, and they are widely used by writers and readers for background knowledge on topics or questions, to help them get up to speed, or to help to set the stage for further research. Even though one might have some understanding of a topic as one begins to try to learn more, finding a literature review can provide access to a lot of work that has already been completed. Of course, one’s critical thinking about what is found in such reviews does not stop after the review is found. The writers of such reviews do commonly cite the resources they used, and so readers can see which resources were used, and they can then evaluate how information from those sources was used. However, it does seem that a reader’s confidence in literature review results can be impacted by seeing what the reviewer did to identify and choose material to use in the review. Readers might ask: how did the reviewer search for materials, and why did he use that approach and not another approach ? Or, why were those resources selected and used, and why not others ? In response, "systematic review" methods have become widely adopted because of the support they give for more reliable literature review results and for the greater confidence readers can have in the conclusions of literature reviews. One goal of this paper to present a possible method (model) for using systematic review methods for topics in religion. Prior to presenting that model, we discuss what literature reviews are, and we see that the new journal Religion Compass is a key resource for literature reviews in religion. We then examine articles from Religion Compass to determine the use of systematic review methods in religion. Next, a model for systematic reviews in religion is presented. We end the paper with a summary and discussion of what we have presented. As a point of background, the ongoing project that served as the basis for information for this paper (and used to develop the model presented) included a “scoping review” process. Scoping is described as including an iterative, exploratory search (a “reconnaissance”) for resources and information (Davis, Drey, & Gould, 2009, p. 1396) . Scoping goals for this project have targeted the topics of literature reviews, systematic reviews and publications in religion that might contain literature reviews. A Systematic Review Model for Relgion 4 What are Literature Reviews and are they used for Topics in Religion ? The goals, rationale, and steps involved with literature reviews are described in books (Hart, 2001; Fink, 2010; Aveyard, 2007), and articles (Boote & Beile, 2005). Web sites have also been developed to support literature review activities. The descriptions below from a sample of such guides indicate what literature reviews involve. 1. Literature reviews involve the “... selection of available documents (both published and unpublished) on the topic, which contain information, ideas, data and evidence written from a particular standpoint to fulfill certain aims or express certain views on the nature of the topic and how it is to be investigated and the effective evaluation of these documents in relation to the research being proposed” (Hart, 2001, p. 13). 2. “A literature review discusses published information in a particular subject area, and sometimes information in a particular subject area within a certain time period. A literature review can be just a simple summary of the sources...(or) a new interpretation of old material or combine new with old interpretations...(or) trace the intellectual progression of the field...” (“Literature Reviews,” n.d.). 3. A literature review is a survey “...of previously published literature on a particular topic to define and clarify a particular problem; summarize previous investigations; identify relations, contradictions, gaps, and inconsistencies in the literature; and suggest the next step in solving the problem” (“APA Databases Methodology,” April, 2012) 4. A literature review is "...a paragraph, section, or entire chapter depending on the nature and length of the publication in which the author identifies and comments on previous attempts to answer the same, or related, research questions" (George, 2008, p. 10). 5. A traditional or narrative “...review critiques and summarizes a body of literature and draws conclusions about the topic in question. The body of literature is made up of the relevant studies and knowledge that address the subject area” (Cronin, Ryan, & Coughlan, 2010, p. 38). 6. “It seems clear that a general definition of a literature review must contain at least two elements: First, a literature review uses as its database reports of primary or original scholarship and does not report new primary scholarship itself.... Second, a literature review seeks to describe, summarize, evaluate, clarify, and/or integrate the content of the primary reports.” (Cooper, 1988, p. 107). As seen above, literature reviews are variously described, and this small set shows a bit of what it is to ‘review a literature’. But also, by looking at these statements a generic traditional literature review model can be constructed that has features that represent a “typical” traditional literature review (TLR). And this kind of model can be used to see if religion publications contain articles that would be seen as A Systematic Review Model for Relgion 5 TLR. Such features can involve a typical goal statement and a set of steps that are involved in the production of TLR; these are listed below. A generic model for traditional literature review (TLR) TLR Goals To construct a discussion about a topic that is based on ‘what has been said before or published’ about that topic. This discussion can aim to be a ‘summary’, a pro and con discussion, an integration of information, a ‘picture’ of the current status, or an ‘answer’, etc. TLR Stages a. Decide on topic or question. b. Gather information on topic “from the literature” (i.e., find relevant sources, identify pertinent information in those sources). c. Use information to “construct a picture”, or to answer a question, etc. d. Create a ‘finished product’, a document (thesis chapter, article, etc.) where: 1. The topic is stated. 2. The developed picture is described. 3. Resources used are cited. Using this generic “model”, publications in religion were examined to see if and how TLR are are found in the religion literature. The following list indicates strategies planned as a means of looking for resources in religion that might contain descriptions of TLR or actual TLR. • Ask for input from faculty and librarians in philosophy and religion • Search article databases (e.g., EBSCO’s ATLA) • Search Google Scholar • Search KentLINK/OhioLINK/WorldCat • Examine descriptions of article contents for encyclopedias (e.g., Encyclopedia of Religion) and other sources (e.g., Oxford Bibliographies Online – Buddhism, etc.) • Search specific journals by hand • Examine Religion Compass articles Work with these possible sources is ongoing, but for this paper it seemed clear that a key resource to look at was the new journal Religion Compass. The reasons for this choice, the methods used and results we found are described below. A key reason for choosing RC is found in the description the editors provide for that publication on their web site. Unique in both range and approach, Religion Compass publishes peer-reviewed surveys of the most important research and current thinking from across the entire discipline. Religion A Systematic Review Model for Relgion 6 Compass guides students, researchers and non-specialist scholars through the accumulating body of literature, and navigates the field by laying out the territory, describing divisions and subdivisions of Religious Studies and identifying the major issues within those sections....Religion Compass provides... A new kind of core content: state-of-the-art surveys of current research discuss the major topics, issues, viewpoints, and controversies within each area of the discipline. Coverage of the entire field highlights connections across sub-disciplines of scholarship in religion ...There is truly no journal in existence like it....a veritable web of topics, issues, viewpoints, and controversies...a diverse array of topics relating to Ancient Near Eastern Studies, Judaic Studies, Christianity, Islamic Studies, New Religious Movements, Indian Traditions, Buddhism, Chinese and Japanese Traditions, African Traditional Religions, Native American Traditions, as well as Theory and Method in the Study of Religion. (“Religion Compass”, n.d.). Note their references to “surveys of the most important research and current thinking”, and to “state-of-the-art surveys of current research”, etc. Comparing these goals of the RC editors with the common descriptions found in the literature about TLR quoted above, and to the “Generic TLR Model” we present above, it seems clear that the RC publication intends to provide precisely the kind of content that should be called TLR in religion. Another level of information showing the presence of TLR in the RC resource is found in the statements of the authors of the articles. Records for all 68 articles from volume 4 of RC were downloaded (on 2/13/12), and from these a set of 10 articles were randomly selected for review. This very small collection was scanned to ‘get an initial picture’ of the “surveys” in that publication. That is, the online HTML abstract and full text for each article was examined to see the author’s stated purpose and to see the author’s presentation of material. Statements taken to represent the author purposes were selected and examined by this paper’s first author, and these stated purposes are listed below. These can be read to see if and how they mirror what is stated by the RC editors, and what is found in the “generic traditional literature review” presented above in this paper. Quoted content was taken from the titles, abstracts or introductory sections of the online articles; and full bibliographic information for these 10 articles is listed numerically in Appendix A. A Systematic Review Model for Relgion 7 1. “ The modest purpose of this essay is to survey exciting new directions in Cusanus research and indicate how these overcome the divisions of earlier interpretations.” 2. “The aim of this essay is to introduce readers to recent work on the concept of ‘civil religion’... The essay reviews the recent contributions of Marcela Cristi and Emilio Gentile. However, it also seeks to introduce readers to some old and recent work on the history of democratic modernity...” 3. “A good deal of important scholarship on early Indian Mahayana Buddhism has been done in recent years....Part 1 of this article surveys and evaluates these recent developments. Part 2 will present a number of new perspectives for future scholarship.” 4. “The intersections between class and religion were of central interest to early social theorists....This paper traces that history and finds evidence of new life in the analysis of class and religion.” 5. “Zaydism: A Theological and Political Survey...Recent scholarship suggests that the origins of Zaydism are considerably more complicated than their portrayal in heresiographical works.” 6. “ While the first part of this article is a general overview of available primary sources on Bible translation into the major Indian languages...In the second part of the article, I review the available secondary sources with special reference to the Tamil language.” 7. “Muslims have an important contribution to make to the growing debate on the environment....This essay is an attempt to explain how Islam defines human connections with the earth and how we may be kinder to it as the source of the generous gifts that sustain all our lives.” 8. “This article contrasts the study of Southeast Asian Buddhist social and political movements and institutions (ecclesiastical, colonial, royal, or governmental) with the study of individuals who are recreating Buddhism outside of the various national Sangha organizations and political structures....” 9. “The Yoruba peoples of present-day south-western Nigeria are the fountainhead of a number of vibrant and growing religious traditions in the Americas...This article surveys the role of Yoruba civilization as the source of these traditions...” 10. “The following article surveys the present discussion and points out new trends as well as major areas of debate: When, and why was the Pentateuch redacted, and how did it come to be accepted as the ‘Torah’? What are the historical and sociological factors underlying this development?...” Looking at the stated purposes of the 10 articles, and looking at the stated purpose of the RC Editors, it does seem that these article goals do mirror (or allow for viewing their purpose as) that of the generic TLR model. Also, as one reads the full articles, the structure of the articles appears to be a A Systematic Review Model for Relgion 8 combination of narration along with the citing of resources used from the literature; thus, a general structure of the article presentations found in RC seemed to be as follows: 1. The author states the question or describes the topic to be addressed/surveyed. 2. The author proceeds to provide the “picture of the topic” that they have constructed from “the literature”. 3. The author cites the sources used in the survey article. These results regarding author purpose statements and evident article structure do seem to show that the content of RC articles mirrors that of a generic traditional literature review model; and so TLR do seem to be found in the religion literature. The next question is this: do TLR in religion exhibit the characteristics of SR ? What are Systematic Reviews and are they used for Topics in Religion ? The introduction to this paper pointed to a development in LR methodology that has been adopted by many disciplines because it provides a framework for more rigorous outcomes for LR projects. Decades ago Glass and his colleagues noted the following seemingly obvious point: “How one searches determines what one finds; and what one finds is the basis of the conclusions of one’s integration (Glass, et al., 1981, p. 61). In that highly cited work, the writers were seeking to develop methods that would improve the integration of published work beyond the results typically found in TLR; and for many years large international organizations (e.g., Campbell Collaboration; Cochrane Collaboration; EPPI-Centre; Centre for Reviews and Dissemination) have worked to address perceived shortcomings of TLR for issues in the health sciences, the social sciences, and for issues related to public policy. From research in the field of bioethics, two sample statements point out that SR methods aim to "...avoid decisions based on an unrepresentative subset of literature relevant to a topic, and to avoid unconscious or conscious "cherry picking" of (use of) publications supportive of a reviewers views" and, conversely, SR methods aim to “enable decisions that are maximally informed and minimally biased.” (Sofaer & Strech, 2011, p. 2). New journals are focused on SR approaches: Research Synthesis Methods and Systematic Reviews; and significant discussions of SR methods are ongoing (Tricco, Tetzlaff, & Moher, 2011). A Systematic Review Model for Relgion 9 As for how to pursue SR, the online “Cochrane Handbook for Systematic Reviews of Interventions” (http://www.cochrane-handbook.org) has the following chapters that represent stages for preparing a systematic review; and the chapters contain considerable detail on the reasoning and steps involved. The chapter headings suggest stages. • Defining the review question and developing criteria for including studies • Searching for studies • Selecting studies and collecting data • Assessing risk of bias in included studies • Analysing data and undertaking meta-analyses • Addressing reporting biases • Presenting results and "summary of findings" tables • Interpreting results and drawing conclusions These stages (and the detailed steps) might be viewed as “a model” for doing SR. And to see if any SR methods are used in religion TLR, we might look at actual TLR publications in religion to see if those steps are being used. To the extent that they were being used, to that extent those publications might be seen as examples of SR in religion. However, a central necessary feature of SR is the reporting of the literature search (LS) methods that are used. LS methods are the steps used to discover the resources and information that is then used in the review. The importance of LS reporting is emphasized in the manuals of the organizations noted above, in articles (Yoshii, Plaut, McGraw, Anderson, & Wellik, 2009), and in the well-regarded PRISMA Statement (http://www.prisma-statement.org). Such reporting is the basis for evaluating the process used by the review writer to develop what they present. Detailed reporting also gives both reader and writer of the reviews a much more solid framework for extending the purpose of the review, or repeating it to see if the “same results” are achieved, or repeating it and using those results with an explicitly different framework for deriving meaning from those results. Detailed reporting is the basis for the confidence that has been expected with systematic reviews results. This LS reporting is central to SR, and so it was decided that the RC articles would be examined for the presence of LS reporting. To the extent that LS reporting was evident, we would view those publications as showing the use of that core part of SR methodology. A Systematic Review Model for Relgion 10 The process we used to evaluate RC articles was as follows. The records for all the RC articles from volumes 5 and 6 had been downloaded on 1/16/12 to create an RC RefWorks database. After both of this paper’s authors reviewed criteria for assessing the evidence of literature search reporting in the articles, they independently examined a set of 20 articles randomly selected from the RC database. Two independent assessors are commonly used as a check on reliability in these kinds of document assessment activities. The criteria for assessing the presence of LS reporting in the religion articles were as follows: the article would need to include language describing strategies such as the use of personal library collections, using computers/databases/online catalogs, contacting experts, using references to related research (e.g., snowballing), handsearching journals, or grey literature searching, etc. Accessing the HTML version of these articles online, each evaluator independently ‘scored’ each of the 20 articles for the presence of LS reporting. That is, if the article contained statements that reported the use of any of these strategies, then the article was judged as “Yes”. Or, the score could be “No”, the article did not report LS steps, or “Maybe” the article reported LS steps. Articles evaluated are in Appendix B. After independent assessments, the authors met to discuss results and resolve any different conclusions regarding the presence of LS reporting. Two articles initially received a score of “Maybe” from one of the authors (see 2. and 4. in Appendix B). Each of these had some discussion that seemed to imply the use of “handsearching” of journals. A third article was judged Maybe and Yes by the authors at first because of discussion indicating the potential of using certain resources for future research on the topic (see 5. in Appendix B). However, both authors agreed that none of these actually presented descriptions of search strategies used to find materials that would have information used in their “survey” articles. These results were viewed as a significant indication that the religion literature does not currently include systematic reviews. A Systematic Review Model for Relgion 11 A Model for Systematic Reviews in Religion This paper has focused on the possible use of SR in religion. Above we discussed and used a 'model' that represents traditional literature reviews TLR, and found that TLR are present in the religion literature. We also used a key part of a generic model of SR; and we saw that SR are not being used in the religion literature. This section describes a kind of generic SR model that could be explored for use across a wide range of ‘religion’ topics, including those that are in the ‘sciences, social sciences, and humanities’. Adapting descriptions of SR found in the SR literature this is what a model for religion might look like: SR Model for Religion General Features A central feature from the beginning of the process is the documentation of all steps used or choices (and the associated reasoning and/or observations for each of those). Also goal or process adjustments might be required throughout; these are documented. Also, much of the SR literature explains the value of having two investigators involved, and so that could be considered. Another significant consideration is the creation of an a priori ‘protocol’ that stipulates goals and actions to be taken for every step in the process; a protocol supports transparency, helps to guard against bias and process subjectivity, and it is expected for those submitting SR to the organizations noted (e.g., Cochrane Collaboration). SR Model for Religion – A Set of Steps 1. Research Topic. Begin to formulate (or actually formulate) and state a focused research topic or question; the SR literature sees focused topics as different from “looking for whatever is related to this topic”. And state the kind of product to be completed; for example, a pro and con discussion, or an integration of information, or a commentary on what ‘is to be found’, etc. Also state the purpose for the project; or, state why or how that completed project might be helpful. This information guides all the steps. 2. Initial scoping searches for information. This is to initially ‘map the area, see what's there, determine the breadth and scope’, etc. Possibly refine, clarity, or focus the topic, or purpose. Develop and A Systematic Review Model for Relgion 12 stipulate what the inclusion/exclusion (IE) criteria will be; these will guide the selection of resources to use and the selection of information from those sources. 3. Search design. Design a comprehensive or targeted search. Record the types of searches to be used, for example: use of personal library collections, computers/databases/online catalogs, contacting experts, using references to related research (e.g., sn owballing), handsearching journals, grey literature searching, etc. Report the reasoning for those strategy choices. Include how and why strategies will be used or not used, including why common, proven strategies are not used if they are not (e.g., see Fehrmann & Thomas, 2011). 4. Do searches. Run and document the search strategies; reproducibility of search (and comprehensiveness) is a key support for reader confidence. Complete an initial screening of results of the different search strategies; and use the 'IE criteria' to identify and retrieve resources (articles, books, etc.) that seem likely to have needed information. Document which sources are selected and why; and document why certain results/hits/sources were not retrieved. 5. Second screening. Using the retrieved resources that passed initial screening, and the IE criteria, decide which items to keep for use. This is a more thorough examination of the resource retrieved; it’s a look at more of the detailed contents. Document which resources you do not use that initially seemed to have value, and why you don't use them. 6. Identify and/or extract useful information. In the resources chosen, identify the specific content that has value for the topic being pursued (e.g., arguments, data, etc.). Document what information you are using, and why. Consider using information tables to chart the information selected (e.g., Garrard, 2007). 7. Construct the arguments, picture, integration, or ‘answer’, etc. that seems to grow out of the information that you have chosen. This step may include explicitly stating how and why you see certain information to be valid, or of a desired quality for the use you are making of it. If not obvious, state how the ‘developed picture’ addresses the goal or purpose that you had for the project ? If the purpose has changed, document and explain this. A Systematic Review Model for Relgion 13 8. Create a 'document' that can be shared that contains the 'results' of steps 1-7. Present this report in a manner that allows others to clearly ‘see’ and ‘reproduce’ steps 1-7. Consider using and/or creating a ‘flow chart’ to show the process used (e.g., see PRISMA materials http://www.prisma-statement.org). Summary and Discussion This paper looked at the topic of literature reviews in religion. We noted that “systematic review” methods have been widely adopted (e.g. in health and social science areas) because such methods enhance the reliability of literature review outcomes and they provide users of literature reviews with a basis for increased confidence in those reviews. Our limited exploration of the religion literature also found that systematic reviews are not used in religion. We described a model that could be explored for using systematic review methods for religion topics, and here suggest that this model or guide would be a new kind of framework for a literature review projects in religion. The perception that the framework would be new is based on the absence of systematic review methods that we found in the results reported above. Moreover, it seems that if writers would use the kind of framework indicated in this model for reviews in religion, then the type reviews produced would provide benefits to readers (and writers) on top of those already available with current literature reviews in religion. For example, the reviews produced might have more reliable conclusions, or support more systematic extension of or expansion on the results of reviews; and they could also support increased confidence in the results of review results. Given these possibilities, the ongoing exploration of how systematic review methods might be further developed and used for topics in religion does seem to have value and is being pursued. A Systematic Review Model for Relgion 14